A Factorized Recurrent Neural Network based architecture for medium to large vocabulary Language Modelling

摘要

Statistical language models are central to many applications that usesemantics. Recurrent Neural Networks (RNN) are known to produce state of theart results for language modelling, outperforming their traditional n-gramcounterparts in many cases. To generate a probability distribution across avocabulary, these models require a softmax output layer that linearly increasesin size with the size of the vocabulary. Large vocabularies need acommensurately large softmax layer and training them on typical laptops/PCsrequires significant time and machine resources. In this paper we present a newtechnique for implementing RNN based large vocabulary language models thatsubstantially speeds up computation while optimally using the limited memoryresources. Our technique, while building on the notion of factorizing theoutput layer by having multiple output layers, improves on the earlier work bysubstantially optimizing on the individual output layer size and alsoeliminating the need for a multistep prediction process.